Towards a Framework for Dependability Benchmarking

The goal of dependability benchmarking is to provide generic ways for characterizing the behavior of components and computer systems in the presence of faults, allowing for the quantification of dependability measures. Beyond existing evaluation techniques, dependability benchmarking must provide a reproducible and cost-effective way of performing this evaluation either as stand alone assessment or more often for comparative evaluation across systems. This paper proposes a framework for defining dependability benchmarks for computer systems, with particular emphasis on COTS and COTS-based systems. The multiple dimensions of the problem are discussed and the framework is presented through a set of dependability benchmarking scenarios, ranging from experimental approaches to benchmarking scenarios where modeling and experimentation are tightly coupled. The main problems and proposals behind each benchmarking scenario are discussed and concrete case studies presented.